Programming by Demonstration with Situated Semantic Parsing

Introduction Programming by Demonstration (PbD) is an approach to programming robots by demonstrating the desired behavior (Billard et al. 2008). Speech is a natural, hands-free way to augment demonstrations with control commands that guide the PbD process. However, existing speech interfaces for PbD systems rely on ad-hoc, predefined command sets that are rigid and require user training (Weiss et al. 2009; Akgun et al. 2012; Cakmak and Takayama 2014). Instead, we aim to develop flexible speech interfaces to accommodate user variations and ambiguous utterances. To that end, we propose to use a situated semantic parser that jointly reasons about the user’s speech and the robot’s state to resolve ambiguities. In this paper, we describe this approach and compare its utility to a rigid speech command interface.